Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Codspeed - performance benchmarks #471

Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open

Codspeed - performance benchmarks #471

wants to merge 17 commits into from

Conversation

tony
Copy link
Member

@tony tony commented Sep 22, 2024

Note

Problem

It's difficult to catch performance degradation or improvements over time, in a PR, etc.

Changes

Add performance benchmarks

TBD

Setup codpseed

Configure on website, set secret, etc.

py(deps[test]) Add pytest-codspeed

See also:

Copy link

codecov bot commented Sep 22, 2024

Codecov Report

Attention: Patch coverage is 83.64780% with 26 lines in your changes missing coverage. Please review.

Project coverage is 64.09%. Comparing base (bc6e897) to head (4ac41d3).

Files with missing lines Patch % Lines
src/libvcs/pytest_plugin.py 80.30% 16 Missing and 10 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #471      +/-   ##
==========================================
+ Coverage   63.85%   64.09%   +0.24%     
==========================================
  Files          40       40              
  Lines        3591     3724     +133     
  Branches      774      790      +16     
==========================================
+ Hits         2293     2387      +94     
- Misses        772      800      +28     
- Partials      526      537      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tony tony force-pushed the pytest-codspeed branch 3 times, most recently from 61f4aa8 to 5ec5337 Compare September 22, 2024 11:24
@tony tony force-pushed the pytest-codspeed branch 5 times, most recently from 9265999 to 99fa170 Compare October 11, 2024 19:00
@tony tony force-pushed the pytest-codspeed branch 2 times, most recently from f649e7f to d0e1548 Compare October 12, 2024 14:53
@tony tony force-pushed the master branch 2 times, most recently from 9b797e7 to bc6e897 Compare October 12, 2024 15:51
tests/sync/test_git.py:11: error: Skipping analyzing "pytest_codspeed.plugin": module is installed, but missing library stubs or py.typed marker  [import-untyped]
tests/sync/test_git.py:11: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
tony added a commit that referenced this pull request Oct 12, 2024
# Problem

Git, Mercurial, and Subversion repositories are unnecessarily reinitialized for each test.

- We're not utilizing session-based scoping.
  - A single initial repo could be created, then copied to [`tmp_path`](https://docs.pytest.org/en/8.3.x/how-to/tmp_path.html#the-tmp-path-fixture) using [`shutil.copytree`](https://docs.python.org/3/library/shutil.html#shutil.copytree) ([source](https://github.com/python/cpython/blob/v3.13.0/Lib/shutil.py#L550-L605)).

Issue #471 highlighted this inefficiency, where benchmarks showed tens of thousands of redundant functional calls.

# Improvement

```
❯ hyperfine -L branch master,pytest-plugin-fixture-caching 'git checkout {branch} && py.test'
Benchmark 1: git checkout master && py.test
  Time (mean ± σ):     32.062 s ±  0.869 s    [User: 41.391 s, System: 9.931 s]
  Range (min … max):   30.878 s … 33.583 s    10 runs

Benchmark 2: git checkout pytest-plugin-fixture-caching && py.test
  Time (mean ± σ):     14.659 s ±  0.495 s    [User: 16.351 s, System: 4.433 s]
  Range (min … max):   13.990 s … 15.423 s    10 runs

Summary
  git checkout pytest-plugin-fixture-caching && py.test ran
    2.19 ± 0.09 times faster than git checkout master && py.test
```

# Changes

## Pytest fixtures overhaul

1. Create a base VCS repo.
2. For subsequent tests, copy and modify from this template.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant